feat: add gunicorn_access_logformat field to Api and Content specs #1578

yaker-gh · 2025-12-30T20:09:19Z

Summary

Adds a new CRD field gunicorn_access_logformat to customize gunicorn's access log format for api and content pods.

Closes #1564

Changes

pulp_types.go

Add GunicornAccessLogformat field to Api struct
Add GunicornAccessLogformat field to Content struct

deployment.go

Modify setEnvVars(): Set PULP_GUNICORN_ACCESS_LOGFORMAT env var with default value if field is empty
Modify pulpcoreApiContainerArgs(): Remove hardcoded --access-logformat from gunicorn ENTRYPOINT array, add --access-logformat "${PULP_GUNICORN_ACCESS_LOGFORMAT}" to exec command
Modify pulpcoreContentContainerArgs(): Add --access-logformat "${PULP_GUNICORN_ACCESS_LOGFORMAT}" to exec command

controller_test.go

Add PULP_GUNICORN_ACCESS_LOGFORMAT to envVarsApi and envVarsContent
Update apiContainers and content container Args to match new shell script behavior

Default format:

pulp [%({correlation-id}o)s]: %(h)s %(l)s %(u)s %(t)s "%(r)s" %(s)s %(b)s "%(f)s" "%(a)s"

This preserves existing behavior when the field is not specified - the default includes the correlation-id header from django_guid.

Usage

spec:
  api:
    gunicorn_access_logformat: '{"remote_ip": "%(h)s", "method": "%(m)s", "path": "%(U)s", "status": "%(s)s"}'
  content:
    gunicorn_access_logformat: '{"remote_ip": "%(h)s", "method": "%(m)s", "path": "%(U)s", "status": "%(s)s"}'

openshift-ci · 2025-12-30T20:09:30Z

Hi @yaker-gh. Thanks for your PR.

I'm waiting for a pulp member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

balasankarc · 2026-01-01T12:47:26Z

@git-hyagi I am not entirely sure why the CI is failing. 🤔 Do you have any idea?

git-hyagi · 2026-01-02T11:41:10Z

Hi @balasankarc!

hmm... in this PR we are trying to define the same default log format for API and content pods, but they use different gunicorn worker classes (which accept different log formats).
For the content pods, we run the aiohttp.GunicornWebWorker. It does not seem to be happy with the format we passed and is crashing:

Error: -30T20:24:29.407798056Z [2025-12-30 20:24:29 +0000] [529] [ERROR] Exception in gunicorn worker
 Traceback (most recent call last):
   File "/usr/local/lib64/python3.11/site-packages/aiohttp/worker.py", line 63, in run
     self.loop.run_until_complete(self._task)
   File "/usr/lib64/python3.11/asyncio/base_events.py", line 654, in run_until_complete
     return future.result()
            ^^^^^^^^^^^^^^^
   File "/usr/local/lib64/python3.11/site-packages/aiohttp/worker.py", line 97, in _run
     access_log_format=self._get_valid_log_format(
                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^
   File "/usr/local/lib64/python3.11/site-packages/aiohttp/worker.py", line 231, in _get_valid_log_format
     raise ValueError(
 ValueError: Gunicorn's style options in form of `%(name)s` are not supported for the log formatting.

What if we define the aiohttp's default log format for the content pods?
https://docs.aiohttp.org/en/stable/logging.html#format-specification

'%a %t "%r" %s %b "%{Referer}i" "%{User-Agent}i"'

balasankarc · 2026-01-02T11:45:51Z

Hi @balasankarc!

hmm... in this PR we are trying to define the same default log format for API and content pods, but they use different gunicorn worker classes (which accept different log formats). For the content pods, we run the aiohttp.GunicornWebWorker. It does not seem to be happy with the format we passed and is crashing:
Error: -30T20:24:29.407798056Z [2025-12-30 20:24:29 +0000] [529] [ERROR] Exception in gunicorn worker
 Traceback (most recent call last):
   File "/usr/local/lib64/python3.11/site-packages/aiohttp/worker.py", line 63, in run
     self.loop.run_until_complete(self._task)
   File "/usr/lib64/python3.11/asyncio/base_events.py", line 654, in run_until_complete
     return future.result()
            ^^^^^^^^^^^^^^^
   File "/usr/local/lib64/python3.11/site-packages/aiohttp/worker.py", line 97, in _run
     access_log_format=self._get_valid_log_format(
                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^
   File "/usr/local/lib64/python3.11/site-packages/aiohttp/worker.py", line 231, in _get_valid_log_format
     raise ValueError(
 ValueError: Gunicorn's style options in form of `%(name)s` are not supported for the log formatting.
What if we define the aiohttp's default log format for the content pods? https://docs.aiohttp.org/en/stable/logging.html#format-specification
'%a %t "%r" %s %b "%{Referer}i" "%{User-Agent}i"'

@git-hyagi Oops. I had figured that out (and commented above) after I pinged you. 😅 Sorry about the noise.

yaker-gh · 2026-01-02T19:09:22Z

@git-hyagi When debugging CI failures like this, what's your usual approach? I spent some time going through the logs but wasn't sure which pods/steps to prioritize.

Is there documentation for the CI pipeline structure, or is this mostly standard k8s deployment patterns I should study up on?

git-hyagi · 2026-01-05T13:04:58Z

Hi @yaker-gh

When debugging CI failures like this, what's your usual approach?

I usually start by checking the failed job, the operator logs, the pods statuses, and Pulp logs. For example, in this case, I saw that some tests worked: it created the Pulp repo/remote/distribution. But, for some reason, it failed to download the container image:

to get a better understanding of why if failed to download the image, I checked the pods statuses (and we probably found a bug in the probes' configuration because all pods were considered READY):

after that I checked the API and content logs:

Is there documentation for the CI pipeline structure, or is this mostly standard k8s deployment patterns I should study up on?

No 😢 we didn't document our CI workflow, most of the steps are shell commands (a mix of make, kubectl, and .sh execution) and they are split as GH jobs, like:

deploy 2 instances of the operator in the same cluster:

pulp-operator/.github/workflows/ci.yml

Lines 76 to 148 in 7c1228e

    
             two-deployments: 
        
               runs-on: ubuntu-latest 
        
               steps: 
        
                 - uses: actions/checkout@v4 
        
                 - name: Setup Go environment 
        
                   uses: actions/setup-go@v5 
        
                   with: 
        
                     go-version-file: 'go.mod' 
        
                     cache: false 
        
                 - name: Install kind 
        
                   run: .ci/scripts/kind_with_registry.sh v1.31.6 
        
                 - name: Install OLM 
        
                   run: | 
        
                     make sdkbin OPERATOR_SDK_VERSION=v1.31.0 LOCALBIN=/tmp 
        
                     /tmp/operator-sdk olm install --version v0.31.0 
        
                   shell: bash 
        
                 - uses: actions/checkout@v4 
        
                 - name: Install Pulp CRD 
        
                   run: make install 
        
                 - name: Build operator image 
        
                   run: make docker-build docker-push IMG=localhost:5001/pulp-operator:dev 
        
                 - name: Build bundle image 
        
                   run: make bundle bundle-build bundle-push BUNDLE_IMG=localhost:5001/pulp-operator-bundle:testing IMG=localhost:5001/pulp-operator:dev 
        
                 - name: Install the operator 
        
                   run: /tmp/operator-sdk run bundle --skip-tls localhost:5001/pulp-operator-bundle:testing 
        
                 - name: Deploy example-pulp and test-pulp 
        
                   run: | 
        
                     kubectl apply -f config/samples/simple-with-reduced-migration-cpu.yaml 
        
                     kubectl apply -f config/samples/simple-test.yaml 
        
                 - name: Check and wait for example-pulp deployment 
        
                   run: kubectl wait --for condition=Pulp-Operator-Finished-Execution pulp/example-pulp --timeout=600s 
        
                 - name: Check and wait for test-pulp deployment 
        
                   run: kubectl wait --for condition=Pulp-Operator-Finished-Execution pulp/test-pulp --timeout=600s 
        
                 - name: Logs 
        
                   if: always() 
        
                   run: .github/workflows/scripts/show_logs.sh --kind 
        
                 - name: OLM Logs 
        
                   if: always() 
        
                   run: | 
        
                     echo ::group::SUB 
        
                     kubectl get sub -oyaml -A 
        
                     echo ::endgroup:: 
        
                     echo ::group::InstallPlan 
        
                     kubectl get ip -A -oyaml 
        
                     echo ::endgroup:: 
        
                     echo ::group::PODS 
        
                     kubectl -nolm get pods 
        
                     kubectl get pods -A 
        
                     echo ::endgroup:: 
        
                     echo ::group::CATALOG-OPERATOR-LOGS 
        
                     kubectl -nolm logs deployment/catalog-operator 
        
                     echo ::endgroup:: 
        
                     echo ::group::OLM-OPERATOR-LOGS 
        
                     kubectl -nolm logs deployment/olm-operator 
        
                     echo ::endgroup:: 
        
                     echo ::group::CATALOGSOURCE 
        
                     kubectl get catalogsource -oyaml -A 
        
                     echo ::endgroup:: 
        
                     echo ::group::CATALOGSOURCE-POD-LOGS 
        
                     kubectl -ndefault logs localhost-5001-pulp-operator-bundle-testing 
        
                     kubectl -ndefault describe pod/localhost-5001-pulp-operator-bundle-testing 
        
                     echo ::endgroup:: 
        
                     echo ::group::CSV 
        
                     kubectl get csv -A -oyaml 
        
                     echo ::endgroup:: 
        
                   shell: bash

deploy with helm:

pulp-operator/.github/workflows/ci.yml

Lines 165 to 215 in 7c1228e

    
             helm: 
        
               runs-on: ubuntu-latest 
        
               strategy: 
        
                 fail-fast: false 
        
               steps: 
        
                 - uses: actions/checkout@v4 
        
                 - uses: ./.github/actions/pre-reqs 
        
                   with: 
        
                     ingress-type: nodeport 
        
                 - name: Build operator container and install CRDs 
        
                   run: | 
        
                     make docker-build install IMG=localhost/pulp-operator:devel 
        
                 - name: Create pulp-operator namespace 
        
                   run: | 
        
                     kubectl create ns pulp-operator-system 
        
                     kubectl config set-context --current --namespace=pulp-operator-system 
        
                 - uses: actions/checkout@v4 
        
                   with: 
        
                     repository: pulp/pulp-k8s-resources 
        
                     ref: main 
        
                 - name: Install helm chart from source 
        
                   run: | 
        
                     helm install --skip-crds --set namespace=pulp-operator-system --set image=localhost/pulp-operator:devel pulp helm-charts/ 
        
                 - uses: actions/checkout@v4 
        
                 - name: Create Pulp Operator CR 
        
                   run: | 
        
                     kubectl apply -f .ci/assets/kubernetes/pulp-admin-password.secret.yaml 
        
                     kubectl apply -f config/samples/simple.yaml 
        
                 - name: Check and wait pulp-operator deploy 
        
                   run: | 
        
                     kubectl wait --for=condition=Ready  $(kubectl get pod  -l app.kubernetes.io/component=operator -oname) 
        
                     kubectl logs -f -l app.kubernetes.io/component=operator -c manager & 
        
                     kubectl wait --for condition=Pulp-Operator-Finished-Execution pulp/example-pulp --timeout=900s 
        
                 - name: Test all components 
        
                   run: | 
        
                     .ci/scripts/pulp_tests.sh -m 
        
                   shell: bash 
        
                   env: 
        
                     PY_COLORS: '1' 
        
                 - name: Logs 
        
                   if: always() 
        
                   run: | 
        
                     .github/workflows/scripts/show_logs.sh 
        
                     helm list

diff configs (like ingress, external db, object storage):

pulp-operator/.github/workflows/ci.yml

Line 252 in 7c1228e

uses: "./.github/workflows/components.yml"

https://github.com/pulp/pulp-operator/blob/main/.github/workflows/components.yml#L6-L87

yaker-gh · 2026-01-05T18:19:30Z

Thanks, @git-hyagi. That's insightful and immensely helpful.

* Pass gunicorn_access_logformat as env var and use in container args * Add gunicorn_access_logformat field to Api and Content specs * Add changelog * Update controller_test.go with new env vars and args * Run `make manifests` and `make bundle` to update CRDs closes pulp#1564

balasankarc · 2026-01-06T06:38:59Z

Tests are passing
Squashed commit
Commit message has reference to issue
Changelog file

I deployed the change in a GKE instance, and everything seems to be working fine. I think this one is good to go.

Thanks @yaker-gh

balasankarc · 2026-01-06T06:39:59Z

/ok-to-test

balasankarc · 2026-01-06T06:48:12Z

/lgtm

controllers/repo_manager/controller_test.go

openshift-ci · 2026-01-06T06:49:47Z

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: balasankarc, yaker-gh
Once this PR has been reviewed and has the lgtm label, please assign ipanova for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Details

Needs approval from an approver in each of these files:

OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

balasankarc · 2026-01-06T07:17:41Z

@git-hyagi We still need your approval. 😁

balasankarc · 2026-01-06T07:46:19Z

We still need your approval. 😁

Spoke too soon. I could merge.

Sorry, GitHub UI is very confusing, and openshift bot added to that. 😅

openshift-ci bot requested review from dkliban and git-hyagi December 30, 2025 20:09

openshift-ci bot added the needs-ok-to-test label Dec 30, 2025

yaker-gh mentioned this pull request Dec 30, 2025

Allow gunicorn log format to be customized #1567

Closed

yaker-gh force-pushed the allow-gunicorn-log-format-to-be-customized branch from 67635e4 to 78cde5b Compare December 30, 2025 20:15

yaker-gh force-pushed the allow-gunicorn-log-format-to-be-customized branch from 78cde5b to 1c0951a Compare January 5, 2026 18:45

openshift-ci bot added ok-to-test and removed needs-ok-to-test labels Jan 6, 2026

openshift-ci bot assigned balasankarc Jan 6, 2026

openshift-ci bot added the lgtm label Jan 6, 2026

balasankarc self-requested a review January 6, 2026 06:48

balasankarc assigned yaker-gh and unassigned balasankarc Jan 6, 2026

balasankarc approved these changes Jan 6, 2026

View reviewed changes

controllers/repo_manager/controller_test.go Show resolved Hide resolved

openshift-ci bot assigned balasankarc Jan 6, 2026

balasankarc merged commit 391d3fa into pulp:main Jan 6, 2026
46 of 48 checks passed

yaker-gh deleted the allow-gunicorn-log-format-to-be-customized branch January 6, 2026 16:35

feat: add gunicorn_access_logformat field to Api and Content specs #1578

feat: add gunicorn_access_logformat field to Api and Content specs #1578

Conversation

yaker-gh commented Dec 30, 2025

Summary

Changes

pulp_types.go

deployment.go

controller_test.go

Usage

Uh oh!

openshift-ci bot commented Dec 30, 2025

Uh oh!

balasankarc commented Jan 1, 2026

Uh oh!

git-hyagi commented Jan 2, 2026

Uh oh!

balasankarc commented Jan 2, 2026

Uh oh!

yaker-gh commented Jan 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

git-hyagi commented Jan 5, 2026

Uh oh!

yaker-gh commented Jan 5, 2026

Uh oh!

balasankarc commented Jan 6, 2026

Uh oh!

balasankarc commented Jan 6, 2026

Uh oh!

balasankarc commented Jan 6, 2026

Uh oh!

Uh oh!

openshift-ci bot commented Jan 6, 2026

Uh oh!

balasankarc commented Jan 6, 2026

Uh oh!

Uh oh!

balasankarc commented Jan 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

yaker-gh commented Jan 2, 2026 •

edited

Loading

balasankarc commented Jan 6, 2026 •

edited

Loading